Experimental constraints on fourth generation quark masses 
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The existing bounds from CDF on the masses of the fourth generation quarks, t' and b' , are 
reexamined. The bound of 256 GeV on the t' mass assumes that the primary decay of the t' is into 
q + W, which is not the case for a substantial region of parameter space. The bound of 268 GeV 
on the b' mass assumes that the branching ratio for b' — > b + Z is very large, which is not only not 
true for much of parameter space, but is never true for b' masses above 255 GeV. In addition, it is 
assumed that the heavy quarks decay within the silicon vertex detector, and for small mixing angles 
this will not be the case. The experimental bounds, including all of these effects, are found as a 
function of the other heavy quark mass and the mixing angle. 
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The question of whether or not there exist quarks 
and leptons beyond the known three generations has 
generated a number of theoretical and experimental 
investigations [H Although direct experimental con- 
straints did not (and do not) rule out a heavy fourth 
generation, until recently electroweak precision data ap- 
peared to disfavour its existence. In addition, there is 
a strong prejudice against quarks and leptons beyond 
the third generation which is usually paraphrased by the 
question: Why is the fourth neutrino so much heavier 
(m U4 > Mz/2) than the other three? Of course, one 
can very well imagine several scenarios in which this can 
"easily" be accomplished since much is yet to be learned 
about neutrino masses. In the end, it will be the verdict 
of experiments which will be the determining factor. 

There is still the question: Why should one bother with 
a fourth generation? There might be several answers to 
that question. First, it is possible that a heavy fourth 
generation might help in bringing the SU(3) ® SU (2) l ® 
U(1)y couplings close to a unification point at a scale 
~ 10 16 in the simplest non-supersymmetric Grand Unifi- 
cation model SU{b) Second, its existence might have 
some interesting connections with the mass of the SM 
Higgs boson [J]. Last but not least, there is no theo- 
retical reason that dictates the number of families to be 
three. We still have no satisfactory explanation for the 
mystery of family replication. 

Recent reexaminations 0, Q of electroweak precision 
data led to the conclusion that the possible existence 
of a fourth generation is not only allowed but its mass 
range is correlated in an interesting way with that of 
the Higgs boson in the minimal Standard Model (SM). 
In the masses of the fourth generation quarks (t' 
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and b') are taken to be bounded from below at around 
258 GeV. This lower bound was taken from CDF, who 
bounded the t' mass@ and the b' mass@- However, CDF 
made a number of assumptions. For instance, it was as- 
sumed that B(b' — ► b+ Z) = 100% which yields a lower 
bound my > 268 GeV. As we will show below, this as- 
sumption is not only unjustified, but is in fact false for 
my > 255 GeV. As a result, the CDF bound weakens 
considerably Q. Furthermore, we show that there exist 
unexplored regions in the mass- mixing angle parameter 
space which could prove very useful to future searches. 

Most of the present mass bounds on the fourth gener- 
ation quarks involved searches done within about 1 cm 
from the beam pipe, inside the silicon vertex detector of 
CDF. The customary focus is on the decay of t' and b' 
into quarks of the first three generations, in particular 
into the 6 quark. As noted above, for example, the most 
recent bound on the b' mass by CDF was obtained by 
searching for the decay mode b' — > b + Z . Similarly, the 
t' search focused on the decay mode t' — > q + W . 

Let us first look at the t' decay. One can consider two 
regions: (I) m t > < my, and (II) my > my. For (I), it is 
obvious that the main decay mode will be t' — > q + W. 
For (II), whether or not t' — > q + W dominates over 
if — ► b' + W depends on the b' mass and the mixing angle 
6bt> ■ In this region, we calculate the branching ratio for 
t' — > q + W as a function of the parameters. If it is lower 
than 60%, the CDF bound will not apply (the choice 
of 60 % is based on viewing the graph in Ref. @ , and 
is somewhat arbitrary without a more detailed analysis- 
since the region shown is on a logarithmic scale, changing 
that number will not noticeably affect the result). If the 
b' mass is between my—Mw and my , then the three body 
decay predominates; whereas if it is lower than my — 
Mw, the two-body decay will dominate. The decay rate 
is given in Refs. [H, [Toj and repeated (in this specific 
context) in equation [52] of Ref. [l[ . 

In addition, even if the t' — > q + W decay does domi- 
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FIG. 1: Bound on the t' mass in the my — sin 2 6 bt i plane. 
The shaded region corresponds to the CDF lower bound of 
256 GeV. 



nate, CDF assumed that it decayed within approximately 
1 centimeter from the beam pipe. For very small mixing 
angles, this will not be the case. Of course, for extremely 
small mixing angles, such that the decay length is greater 
than about 3 meters, the t' will appear to be a stable par- 
ticle and can be ruled out (above some mass) by stable 
quark searches. 

All of these results are plotted in Fig. 1, where we 
plot the bound on the t 1 mass in the my — sin 2 9bt> 
plane. There are three distinct regions. The shaded re- 
gion above and to the right of the curve with m t < = 256 
GeV represents the CDF lower bound on m# ■ In this re- 
gion, the CDF bound applies. Below the shaded region, 
the curves correspond to the new lower bound on 
from CDF based on the requirement that the t' — > q + W 
decay is dominant. These curves all end to the left 
at sin 2 Obt' ~ 6 x 10~ 15 . This corresponds to a decay 
length of approximately 1 cm. (Let us recall that present 
searches are sensitive to decays which occur very close 
to the beam pipe to a distance of about 1 cm.) To the 
left of those curves lies an unexplored window situated 
between sin 2 B w ~ 6 x KT 15 and sin 2 6 W ~2x 1CT 17 , 
corresponding to a distance of roughly 1 cm out to 3 m. 
The far-left of the plot represents the constraint coming 



FIG. 2: B(b' -> b + Z) as a function of My for various M t , 



from the search[llj for a stable t' (at distances greater 
than approximately 3 m). 

For the bounds on the b' mass, CDF assumed that 
the branching ratio B(b' — > b + Z) was 100 percent. In 
Fig. 2, we plot the branching ratio B(b' — > b + Z) as 
a function of my for different values of mp . Here we 
use the results of [13] where the assumption sinObt' = 
— sin 9 t b> = x was made resulting in a GIM cancellation 
when m t ' ~ m t . Furthermore, as in we will assume 
that | sin 9 c y \ < x 2 so that the decay of b' into "lighter" 
quarks will be mainly into the t quark. Note that this 
assumption may not be justified, and if it is false the 
branching ratio will be even lower, weakening the CDF 
bound even further. The decay into t is three-body for 
my < m t + mw and two-body otherwise. This is to 
be compared with b' — > b + Z. This analysis has been 
performed in (Table I) and Q (Fig. 14). 

The results are shown in Fig. 2. It can be seen that 
B{b' — > b+ Z) < 100% for a wide range of b' mass above 
200 GeV. Note that the bound of 268 GeV on my as- 
suming B(V -> b+ Z) = 100% does not hold. As my 
gets above 200 GeV, the decay mode &'—>•£+ W* begins 
to be comparable with the mode b' — > b + Z and starts 
to dominate for my > 250 GeV 12]. In particular, for 
my > 255 GeV, the decay b' — » t + W will be into real 
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FIG. 3: Bound on the b' mass in the m t i — sin 2 8 t y plane. 

particles, and thus this decay will always dominate. The 
CDF bound can thus never exceed 255 GeV. Using Fig. 
2, we estimate the acceptance for b' ^ b + Z &s had been 
done by CDF Q. The results are then used as inputs 
into our analysis of the bounds on the b' mass. 

The b' decay is treated in a similar fashion. Again, we 
subdivide the decay into two regions: (I) my < m t >, and 
(II) mb> > m t i . Different curves in the m t ' — sin 2 Otw 
plane correspond to different values of my for which b' 
does not decay into t' . This is shown in Fig. 3. Here, we 
take into account the value of B(b' — > b + Z) (denoted 
by (3 in Q) as obtained from Fig. 2 for a given m t / and 
my ■ We then use this number to obtain the acceptance 
as given by Q which is scaled by a factor 1 — (1 — j3) 2 . Dif- 



ferent values of rrib> for different curves in Fig. 3 reflect 
this acceptance constraint. We also show an "unexplored 
region" similar to that shown in Fig. 1 for decays occur- 
ing between 1 cm and 3 m, as well as the region where b' 
is "stable" . This unexplored region is not vertical, as in 
Fig. 1, since the rate for b' — > b + Z is very sensitive to 
the t' mass. 

In summary, we reexamined the experimental bounds 
on the masses of the fourth generation quarks: the t' 
and V quarks. We divide the search into three distance 
regions as measured from the center of the beam pipe: 
1) d = cm to ~ 1 cm, 2) d ~ 1 cm to 3 m and 3) 
d>3m. The first region is one where most searches at 
the Tevatron have been performed. We have computed 
the lower bounds on the t' and b' masses under the re- 
quirement that t' and b' decay primarily into quarks of 
the first three generations as shown in Fig. 1 and Fig. 
3. For b' , we found that the CDF lower bound on its 
mass can never exceed 255 GeV, contrary to an earlier 
claim of 268 GeV which had made use of the assumption 
B(b' — > b + Z) = 100% and which is not correct when the 
b' mass exceeds 200 GeV. For t' , bounds are shown, start- 
ing with the CDF bound 256 GeV. Region (3) (greater 
than 3 m) is bounded by searches for stable quarks. Re- 
gion (2) (between I cm and 3 m) is unexplored and cor- 
responds to a range of mixing angle sin 2 9bv ~ 6 x I0~ 15 
and sin 2 Ow ~ 2 x 10 -17 . Such a small mixing angle 
might seem unlikely, but it could arise very naturally 
in a 3 + 1 scenario. For example, if one simply had a 
Zi symmetry in which the fourth generation fields were 
odd and all other fields were even, then the mixing angle 
would vanish. However, discrete symmetries will gener- 
ally be broken by Planck mass effects, which can lead to 
sin 2 9bt' of Mw/Mpi ~ 10~ 17 . Thus, such a small mix- 
ing angle could be natural, and we urge our experimental 
colleagues to explore this region. If the fourth generation 
quarks are indeed found in this region, it would shed light 
on the question of family replication. 
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